Skip to content

Honor custom item serializers in typed NumPy arrays - #296

Closed
pengpengyi92 wants to merge 1 commit into
Point72:mainfrom
pengpengyi92:codex/fix-ndarray-item-serialization-20261114
Closed

pengpengyi92 wants to merge 1 commit into
Point72:mainfrom
pengpengyi92:codex/fix-ndarray-item-serialization-20261114

Conversation

@pengpengyi92

@pengpengyi92 pengpengyi92 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Honor the declared custom item serializer when serializing typed NDArray
object arrays. Addresses #58, reported by @NeejWeej.

The current .tolist() -> any_schema path loses the item's declared
Pydantic serialization schema. This can raise PydanticSerializationError
for custom objects, or serialize ccflow.Enum as values rather than names.

This change uses a rank-specific, cached TypeAdapter for arrays whose
items actually retain their declared custom-schema type. It forwards
serialization mode, filters, aliases and context. Untyped arrays and native
NumPy dtypes keep the existing path. NumPy-coerced enum values also keep the
existing path: numbers must not be sent to a serializer expecting objects.
Validation and NumPy dtype conversion are intentionally unchanged.

Regression tests cover 1D/2D/3D and empty object arrays, Python vs JSON mode,
index filters, optional fields, model aliases/filters, context, enum names,
untyped objects and the coerced-IntEnum boundary. No dependency/version
changes; the original MIT notices in these two files are preserved.

Type of Change

  • Bug fix

Local validation

Windows 11, Python 3.12.14. Base: 83f2b0ac7128c08c92f2b6fd60f58723ac47635a.
These are scoped local results, not a claim of complete CI or coverage.

python -m pytest ccflow/tests/exttypes/test_pydantic_numpy.py \
  ccflow/tests/result/test_numpy.py ccflow/tests/enums \
  ccflow/tests/test_context.py -q -o addopts=
python -m ruff check ccflow
python -m ruff format --check ccflow
  • Pydantic 2.14.3 / NumPy 2.5.4: 53 passed, 1 skipped; the NDArray module
    accounts for 21 passing cases. The context/enum selection has an existing
    environment-dependent skip. Ruff check and format check pass (208 files).
  • Pydantic 2.11.10: same selection 53 passed, 1 skipped.
  • Negative control: the candidate NDArray tests run against the clean base
    yield 8 failed, 13 passed. The base's original 7 NDArray tests pass.
  • Actual CSP 0.18.0 / NumPy 1.26.4 / PyArrow 23.0.1: the NDArray doesn't handle types with custom core schemas properly #58 enum example
    fails on the base and passes with this change. Candidate NDArray tests plus
    that separate example: 22 passed. CSP 0.19.1: 22 passed, preserving
    integer output for values already coerced by NumPy.
  • Pydantic 2.6.4 is not green: the original test_union_field fails on
    both the clean base and candidate. Candidate: 19 passed, 1 failed,
    1 skipped (context API unavailable); base: 6 passed, 1 failed. This PR
    does not attempt an unrelated union-validation change.

The native CSP check uses the issue's class-based Enum example with
TypeAdapter(NDArray[Side]). Expected output is enum names on CSP 0.18,
and the unchanged numeric output after NumPy coercion on CSP 0.19.

Not run: full make test, cross-platform CI, optional-service suites or
performance benchmarks. Adapter caching is not a measured speedup claim.

Checklist

  • Full make lint (Ruff commands above pass; other tools not run)
  • Full make test (scoped results and inherited lower-version failure above)
  • New regression tests added
  • No version bump or unrelated documentation changes needed for this fix

Prepared with OpenAI Codex assistance. Local tests and clean-base controls
were actually executed. DCO sign-off was explicitly authorized by the author
for this contribution only.

Upstream CI readback - 2026-11-14 (Asia/Shanghai)

For head 1f1afbc6ba87707fc915d2e1e4913aa7e84c4110, all 16 Linux/macOS
build-matrix jobs in run 43486775195
succeeded, together with DCO and both Codecov checks (19 successful checks
observed). This is separate from the scoped Windows runs above and does not
claim maintainer approval or merge.

Signed-off-by: Pengyi Peng <pengpengyi92@users.noreply.github.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.08738% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.86%. Comparing base (83f2b0a) to head (1f1afbc).

Files with missing lines Patch % Lines
ccflow/tests/exttypes/test_pydantic_numpy.py 96.38% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #296      +/-   ##
==========================================
+ Coverage   93.85%   93.86%   +0.01%     
==========================================
  Files         205      205              
  Lines       22944    23043      +99     
  Branches     1511     1517       +6     
==========================================
+ Hits        21533    21629      +96     
- Misses       1120     1122       +2     
- Partials      291      292       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timkpaine

Copy link
Copy Markdown
Member

Thanks for the PR, i've consolidated several open ones in #299

@timkpaine timkpaine closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants